Assessing calibration in an external validation study.

Evaluating a prediction model using a separate data set from which the model was developed is a crucial step in assessing its predictive performance, often referred to as external validation [1]. The recent study by Tetreault et al. [2] modified their previous prediction model by omitting one of the predictors and then refitting the model on the original development data from 12 sites from North America [3]. The modified prediction model was subsequently evaluated on a larger international cohort from the AOSpine CSM-I trial [4]. Although it is encouraging to see authors carrying out such external validation studies, there are concerns in the analysis that need highlighting. It is widely accepted that the two main characteristics to report when evaluating the performance of a prediction model are discrimination and calibration [5]. In the study by Tetreault et al. [1], calibration has been incorrectly evaluated. The authors have presented the traditional and commonly seen plot of predictions and observed outcomes, by ranking and grouping individuals (47 groups of size 10), and calculating the mean observed outcome against the mean predicted probability. However, the authors have then incorrectly fit a linear regression line to the 47 points and examined whether the resulting intercept and slope are noticeably different from 0 and 1, respectively. The arbitrary creation of groups of size 10 will affect these estimates of the slope and intercept, and different values will result if groups of 5 or 20 were made. The correct approach would be to calculate the slope and intercept fitting the linear predictor (LP; LP51.59 0.81Pþ0.19mJOAþ0.91IG 0.69S 0.27DS) calculated for all individuals in the validation data set as the only predictor in a logistic regression model: log odds (outcome)5aþb LP [6]. A second concern is that the authors refit the model on the validation data set and then made a judgment on the similarity of the regression coefficients. This does not constitute an assessment of validation and provides no useable information on the performance of the prediction model [6]. Validation concerns only the performance evaluation of the prediction model in new data. The Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) Initiative (www.tripod-statement.org) recently published the TRIPOD reporting guideline for clinical prediction models, where key issues in the development and validation of a prediction model are discussed. The TRIPOD guideline is similar to other well-known reporting guidelines (eg, CONSORT, STROBE, PRISMA) designed to help authors, peer reviewers, and journal editors in ensuring that the essential items describing the development or validation of a clinical prediction model are clearly reported [5]. Accompanying the reporting guideline is an extensive Explanation & Elaboration article describing the rationale for the checklist item but also highlighting many methodologic considerations when developing or validating a clinical prediction model [6].